{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 词向量\n",
    "#### 学习目标\n",
    "\n",
    "- 学习词向量的概念\n",
    "- 用Skip-thought模型训练词向量\n",
    "- 学习使用PyTorch dataset和dataloader\n",
    "- 学习定义PyTorch模型\n",
    "- 学习torch.nn中常见的Module\n",
    "    - Embedding\n",
    "- 学习常见的PyTorch operations\n",
    "    - bmm\n",
    "    - logsigmoid\n",
    "- 保存和读取PyTorch模型\n",
    "\n",
    "- 第二课使用的训练数据可以从以下链接下载到。\n",
    "    - 链接:https://pan.baidu.com/s/1tFeK3mXuVXEy3EMarfeWvg 密码:v2z5\n",
    "\n",
    "#### 内容\n",
    "\n",
    "- 在这一份notebook中，我们会（尽可能）尝试复现论文[Distributed Representations of Words and Phrases and their Compositionality](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)中训练词向量的方法. 我们会实现Skip-gram模型，并且使用论文中noice contrastive sampling的目标函数。\n",
    "\n",
    "- 这篇论文有很多模型实现的细节，这些细节对于词向量的好坏至关重要。我们虽然无法完全复现论文中的实验结果，主要是由于计算资源等各种细节原因，但是我们还是可以大致展示如何训练词向量。\n",
    "\n",
    "- 以下是一些我们没有实现的细节\n",
    "    - subsampling：参考论文section 2.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.utils.data as tud\n",
    "from torch.nn.parameter import Parameter\n",
    "\n",
    "from collections import Counter\n",
    "import numpy as np\n",
    "import random\n",
    "import math\n",
    "\n",
    "import pandas as pd\n",
    "import scipy\n",
    "import sklearn\n",
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# 为保证实验结果可以复现，我们经常会把各种random seed固定在某一个值\n",
    "SEED = 2019\n",
    "random.seed(SEED)\n",
    "np.random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.manual_seed(SEED)\n",
    "    \n",
    "# 设定一些超参数\n",
    "K = 100         # 负样本数量\n",
    "C = 3           # word2vec窗口大小（半径）\n",
    "EPOCHS = 2      # 跑完一次全量数据为一次EPOCHS\n",
    "MAX_VOCAB_SIZE = 30000    # 词典容量，即有3万个单词\n",
    "BATCH_SIZE = 128          # 每批次数据大小\n",
    "LEARNING_RATE = 0.2       # 初始学习率\n",
    "EMBEDDING_SIZE = 100      # 词向量维度\n",
    "\n",
    "\n",
    "TRAIN_FILE = \"data/text8/text8.train.txt\"\n",
    "EVAL_FILE = \"data/text8/text8.dev.txt\"\n",
    "TEST_FILE = \"data/text8/text8.test.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenize函数，把一篇文本转化成一个个单词\n",
    "def word_tockenize(text):\n",
    "    return text.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据预处理（文本处理）\n",
    "- 从文本文件中读取所有的文字，通过这些文本创建一个vocabulary\n",
    "- 由于单词数量可能太大，我们只选取最常见的MAX_VOCAB_SIZE个单词\n",
    "- 我们添加一个UNK单词表示所有不常见的单词\n",
    "- 我们需要记录单词到index的mapping，以及index到单词的mapping，单词的count，单词的(normalized) frequency，以及单词总数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_preprocess(txt_file):\n",
    "    with open(txt_file, \"r\") as f:\n",
    "        text = f.read()\n",
    "    text = [w for w in word_tockenize(text.lower())]\n",
    "    # 词频最大的MAX_VOCAB_SIZE-1个单词，剩下一个留给”<unk>\"\n",
    "    vocab = dict(Counter(text).most_common(MAX_VOCAB_SIZE-1)) \n",
    "    # 其他未在vocab中的单词用<unk>代替，并计算unk的词频，添加到vocab字典中\n",
    "    vocab[\"<unk>\"] = len(text) - np.sum(list(vocab.values()))\n",
    "    \n",
    "    idx2word = [word for word in vocab.keys()]  \n",
    "    word2idx = {word: i for i, word in enumerate(idx2word)}\n",
    "    \n",
    "    # 负样本采样\n",
    "    # 每个单词出现的数量列表\n",
    "    word_counts = np.array([count for count in vocab.values()], \n",
    "                           dtype=np.float32)\n",
    "    word_freqs = word_counts / np.sum(word_counts)\n",
    "    word_freqs = word_freqs ** (3./4.)\n",
    "    word_freqs = word_freqs / np.sum(word_freqs)\n",
    "    vocab_size = len(idx2word)\n",
    "    print(vocab_size)\n",
    "    return text, idx2word, word2idx, word_freqs, word_counts, vocab_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30000\n"
     ]
    }
   ],
   "source": [
    "text, idx2word, word2idx, word_freqs, word_counts, VOCAB_SIZE = text_preprocess(TRAIN_FILE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the', 'of', 'and', 'one', 'in', 'a', 'to', 'zero', 'nine', 'two']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx2word[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实现DataLoader\n",
    "一个dataloader需要以下内容：\n",
    "\n",
    "- 把所有text编码成数字，然后用subsampling预处理这些文字。\n",
    "- 保存vocabulary，单词count，normalized word frequency\n",
    "- 每个iteration sample一个中心词\n",
    "- 根据当前的中心词返回context单词\n",
    "- 根据中心词sample一些negative单词\n",
    "- 返回单词的counts\n",
    "\n",
    "这里有一个好的tutorial介绍如何使用[PyTorch dataloader](https://pytorch.org/tutorials/beginner/data_loading_tutorial.html).\n",
    "为了使用dataloader，我们需要定义以下两个function:\n",
    "\n",
    "- ```__len__``` function需要返回整个数据集中有多少个item\n",
    "- ```__get__``` 根据给定的index返回一个item\n",
    "\n",
    "有了dataloader之后，我们可以轻松随机打乱整个数据集，拿到一个batch的数据等等。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 理解torch.multinomial 采样函数\n",
    "--------------\n",
    "- torch.multinomial(input, num_samples, replacement=False, out=None) → LongTensor\n",
    "    - 作用是对input的每一行做num_samples次取值，输出的张量是每一次取值时input张量对应行的**下标**。\n",
    "    - 输入是一个input张量，一个取样数量可以是列表，和一个布尔值replacement。\n",
    "    - input张量可以看成一个权重张量，每一个元素代表其在该行中的权重。如果有元素为0，那么在其他不为0的元素\n",
    "    - 被取干净之前，这个元素是不会被取到的。\n",
    "    - num_samples是每一行的取值次数，该值不能大于每一样的元素数，否则会报错。\n",
    "    - replacement指的是取样时是否是有放回的取样，True是有放回，False无放回。\n",
    "----\n",
    "- 官方例子\n",
    "```python\n",
    "weights = torch.tensor([0, 0.8, 2, 0], dtype=torch.float) # create a tensor of weights\n",
    "print(torch.multinomial(weights, 2))\n",
    "# tensor([2, 1])\n",
    "print(torch.multinomial(weights, 4, replacement=True))\n",
    "# tensor([2, 2, 1, 2])\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class WordEmbeddingDataset(tud.Dataset):\n",
    "    def __init__(self, text, word2idx, idx2word, word_freqs, word_counts):\n",
    "        super(WordEmbeddingDataset, self).__init__()\n",
    "        # 文档编码\n",
    "        self.text_encoded = [word2idx.get(t, VOCAB_SIZE-1) for t in text]\n",
    "        self.text_encoded = torch.Tensor(self.text_encoded).long()\n",
    "\n",
    "        self.word2idx = word2idx\n",
    "        self.idx2word = idx2word\n",
    "        self.word_freqs = torch.Tensor(word_freqs)\n",
    "        self.word_counts = torch.Tensor(word_counts)\n",
    "        \n",
    "    def __len__(self):\n",
    "        ''' \n",
    "        返回整个数据集（所有单词）的长度\n",
    "        '''\n",
    "        return len(self.text_encoded)\n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "        ''' \n",
    "        这个function返回以下数据用于训练\n",
    "            - 中心词\n",
    "            - 这个单词附近的(positive)单词\n",
    "            - 随机采样的K个单词作为negative sample\n",
    "        '''\n",
    "        center_word = self.text_encoded[idx]\n",
    "        pos_indices = list(range(idx-C, idx))+list(range(idx+1, idx+C+1))\n",
    "        # 返回除中心词之外的附近的单词的索引\n",
    "        pos_indices = [i % len(self.text_encoded) for i in pos_indices]\n",
    "        # 周围词编码列表\n",
    "        pos_words = self.text_encoded[pos_indices]\n",
    "        # 多项式分布采样\n",
    "        neg_words = torch.multinomial(self.word_freqs, K * pos_words.shape[0], True)\n",
    "        \n",
    "        return center_word, pos_words, neg_words "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建Dataset与Dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = WordEmbeddingDataset(text, word2idx, idx2word, word_freqs, word_counts)\n",
    "dataloader = tud.DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(428) torch.Size([128])\n",
      "tensor([13053,   267,   314,     1,     0,   113]) torch.Size([128])\n",
      "tensor([ 9576,  2082,   279, 21829,  2247,   953,  3666, 23787,  1562,   465,\n",
      "          408, 11302,  1286,     7,   546,     1,    39,  2556,    16,  6145,\n",
      "         6253,    25,   476,   271,    16,  1830,    28,    12,  1607,    23,\n",
      "         7888,     0,  7863,   331,   521,   406,  6392, 12881,  5695,  5322,\n",
      "         2289,   697, 26196,  1396, 13726,  7781,    19, 21110,    60,    90,\n",
      "           15,   776, 18314,  2433,    19, 12410,  9995, 16336, 29999,  1439,\n",
      "         2480, 17925,    14,     2,  3978,  1225,   349,  1537,  2450,  1053,\n",
      "        26771,  2577,     0,  4777,  1503,  6077,   601,   241,  1621,  8592,\n",
      "           11, 24580,   103,   557,  9434,  9990,    15,     6, 27198,    32,\n",
      "          926,  2860,   251,  2720,   196,    19, 10197,  3980,  5902,   117,\n",
      "            1, 11078,  1063,   912,    26,    25,  7395,    43,  1201,   368,\n",
      "          645,   761,    59, 26620, 25032,  6552, 29999,  5468, 14295,     0,\n",
      "         2328,   817,    30,    37, 16986,   184,    31, 17739,  4321, 18647,\n",
      "          626,   162, 16649,  4789,  1180,     1, 16826,   140,   224,  7348,\n",
      "         4979, 11255,  7222, 21236,  3563,  1613,  1126,  4796,  1260,  1266,\n",
      "           83, 15459,  1384, 23111,    12,  2882,  3453, 21782,   326,     1,\n",
      "        24668,    10,  4514,   177,  1547, 28461,    60,   571,   329,     4,\n",
      "        15079,  5239,  7424,  5960,  1576, 17409,  1876, 11498,  1212,   554,\n",
      "          417,  2810, 28861,   463,  2178,     0,    21,  3636,  1575,   892,\n",
      "         1427, 29811,  1267,   462,   244,    22,  6712,     6,   903,  1686,\n",
      "         2368,    38,  4260, 27994,    18, 13272,   503,     1, 12269,  1584,\n",
      "          124,     0, 10804,  3146,  8851,  2009,  7989,  9000,    30, 11973,\n",
      "            5,  1957,   126,  2184,     2,    54,  9716, 26132,    14,    12,\n",
      "          670, 18220,    49,     5,     6,  8363,  2053, 11037,  5077,    20,\n",
      "         2469, 15912,   748,  4698,   191,    58,  5142,  8578,  7576,   236,\n",
      "         5860,     4,    15, 25395,  3211,    54,   869,     1,   728,     5,\n",
      "         5996,    76,  1113,  2347,  1564,  8866,  6272,    73, 15429,    45,\n",
      "            1,   582,    51, 11152,   873,   153,   338,  7185,    44,    69,\n",
      "          738,  6127, 29999, 24991,  1025,    17,  3745,  2386,  9295, 10882,\n",
      "           59,  9255,   738, 12219,   822,   886,    53,   103,   572,   215,\n",
      "        29999,  2578,  2486,   385,   964,  3356,  3757,  7669,    90,  1982,\n",
      "          891,   236,  4075, 28148,  3235,  8384,    18, 11066,  3144,  2273,\n",
      "           23, 19070, 18571,  9719,    78,  4364,  4607,   999,     1,   291,\n",
      "           35,  4820, 21795,  1066, 22590,  2197,   664,  7210,    29,  1039,\n",
      "        23990,  5002,   508,   860,   242, 27623, 29999,    11, 12204,  2171,\n",
      "         4983,  1646,  1928,     9,   243,   496,    56,   335, 10833,     2,\n",
      "         3567,  1767,   135,  3903,   372,    25,     0,   903,  8868,   444,\n",
      "         2430,  8395,  2152, 10763,    23,  3246,    10, 14784,  2661, 17631,\n",
      "         2893,  1268,  7852,  4110,  2050, 23244,  7547,  4588,  4102,  6179,\n",
      "         9784,  1303,  3242,     9,    92, 13216, 29048,  5437, 20718,   660,\n",
      "          260, 20686,   487,  8095,   342,  7475,  2452,   120, 15514,  8295,\n",
      "           26,   130,  2568, 16588, 15441,  1661,    94,    54,  3798,  4073,\n",
      "        14964,  4568,  3092,   379,  1101, 28703,    24, 22072,  9952, 25327,\n",
      "        12018,  2463,    23,    18,  1454,    25,    89,  1265,  4005, 27452,\n",
      "         4242,    11,   210, 15166, 23341,  1768,    17,    48,   675,  7678,\n",
      "          404,  1432,  6750,  4227,   232, 21635, 16438,     3,     4, 17153,\n",
      "         1275,  1419,  1541,  1904,  4337,  6846,  1808,  2432,  8223, 13159,\n",
      "         1908,  6174,  2304,   986,     2,    15,  3460, 11839,   431,   248,\n",
      "        24557,   101,     1, 13539,  3088, 25414, 17546,  4321,   512,  1326,\n",
      "         2425,   188,  1193,   500,    20,   762,  3018,  9416,   363,    19,\n",
      "         5139, 15786,  4415,     2,  1486,   927,  2003,    44,   107,    72,\n",
      "         1645,  6218,     7,   782, 15815,    16,    61,  6174,    18,    90,\n",
      "        18594,  1550, 15072,  8540,   120,    22, 12451,  5243,  1923,  1293,\n",
      "          305,  1267,  1434,  6581,  2168,    22, 24760,   658,  1668,   738,\n",
      "         2959,    10,  3704, 25956,   382,  4705,    31,  7013,    32,     3,\n",
      "           20, 22645, 22722, 14006,  6683,   288,    74,  3766,  8361,  8454,\n",
      "          117,  3125,    35, 13420,     4,  7523, 13239,    16,  4794,  3857,\n",
      "           39, 29999, 12271,  3992,   320, 18459, 20338,   173,  6212,  2051,\n",
      "         5214,  4421, 28549,   730,    48,   470,  2131,  1405,   383,  1205,\n",
      "          274, 19799,   359, 29999,  3112,  9795, 11527,  5090,  7965, 24565]) torch.Size([128])\n",
      "--------------------\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "for center_word, pos_words, neg_words in dataloader:\n",
    "    print(center_word[0], center_word.size())\n",
    "    print(pos_words[0], center_word.size())\n",
    "    print(neg_words[0], center_word.size())\n",
    "    print(\"-\"*20)\n",
    "    i += 1\n",
    "    if i == 1:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义pytorch 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EmbeddingModel(nn.Module):\n",
    "    def __init__(self, vocab_size, embed_size):\n",
    "        ''' \n",
    "        初始化输入和输出embedding\n",
    "        '''\n",
    "        super(EmbeddingModel, self).__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embed_size = embed_size\n",
    "        \n",
    "        # 两个都初始化的好处？\n",
    "        initrange = 0.5 / self.embed_size\n",
    "        self.in_embed = nn.Embedding(self.vocab_size, self.embed_size, sparse=False)\n",
    "        self.in_embed.weight.data.uniform_(-initrange, initrange)\n",
    "        \n",
    "        self.out_embed = nn.Embedding(self.vocab_size, self.embed_size, sparse=False)\n",
    "        self.out_embed.weight.data.uniform_(-initrange, initrange)\n",
    "        \n",
    "    def forward(self, input_labels, pos_labels, neg_labels):\n",
    "        '''\n",
    "        input_labels: 中心词, [batch_size]\n",
    "        pos_labels: 中心词周围 context window 出现过的单词 [batch_size * (window_size * 2)]\n",
    "        neg_labelss: 中心词周围没有出现过的单词，从 negative sampling 得到 \n",
    "                    [batch_size, (window_size * 2 * K)]\n",
    "        \n",
    "        return: loss, [batch_size]\n",
    "        '''\n",
    "        \n",
    "        batch_size = input_labels.size(0)\n",
    "        \n",
    "        input_embedding = self.in_embed(input_labels) # B * embed_size\n",
    "        pos_embedding = self.out_embed(pos_labels) # B * (2*C) * embed_size\n",
    "        neg_embedding = self.out_embed(neg_labels) # B * (2*C * K) * embed_size\n",
    "      \n",
    "        log_pos = torch.bmm(pos_embedding, input_embedding.unsqueeze(2)).squeeze() # B * (2*C)\n",
    "        log_neg = torch.bmm(neg_embedding, -input_embedding.unsqueeze(2)).squeeze() # B * (2*C*K)\n",
    "\n",
    "        log_pos = F.logsigmoid(log_pos).sum(1)\n",
    "        log_neg = F.logsigmoid(log_neg).sum(1) # batch_size\n",
    "       \n",
    "        loss = log_pos + log_neg\n",
    "        \n",
    "        return -loss\n",
    "    \n",
    "    def input_embeddings(self):\n",
    "        return self.in_embed.weight.data.cpu().numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = EmbeddingModel(VOCAB_SIZE, EMBEDDING_SIZE)\n",
    "model = model.to(device)\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=LEARNING_RATE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(filename, embedding_weights): \n",
    "    if filename.endswith(\".csv\"):\n",
    "        data = pd.read_csv(filename, sep=\",\")\n",
    "    else:\n",
    "        data = pd.read_csv(filename, sep=\"\\t\")\n",
    "    human_similarity = []\n",
    "    model_similarity = []\n",
    "    for i in data.iloc[:, 0:2].index:\n",
    "        word1, word2 = data.iloc[i, 0], data.iloc[i, 1]\n",
    "        if word1 not in word_to_idx or word2 not in word_to_idx:\n",
    "            continue\n",
    "        else:\n",
    "            word1_idx, word2_idx = word_to_idx[word1], word_to_idx[word2]\n",
    "            word1_embed, word2_embed = embedding_weights[[word1_idx]], embedding_weights[[word2_idx]]\n",
    "            model_similarity.append(float(sklearn.metrics.pairwise.cosine_similarity(word1_embed, word2_embed)))\n",
    "            human_similarity.append(float(data.iloc[i, 2]))\n",
    "\n",
    "    return scipy.stats.spearmanr(human_similarity, model_similarity)# , model_similarity\n",
    "\n",
    "def find_nearest(word):\n",
    "    index = word_to_idx[word]\n",
    "    embedding = embedding_weights[index]\n",
    "    cos_dis = np.array([scipy.spatial.distance.cosine(e, embedding) for e in embedding_weights])\n",
    "    return [idx_to_word[i] for i in cos_dis.argsort()[:10]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练模型\n",
    "-----\n",
    "- 模型一般需要训练若干个epoch\n",
    "- 每个epoch我们都把所有的数据分成若干个batch\n",
    "- 把每个batch的输入和输出都包装成cuda tensor\n",
    "- forward pass，通过输入的句子预测每个单词的下一个单词\n",
    "- 用模型的预测和正确的下一个单词计算cross entropy loss\n",
    "- 清空模型当前gradient\n",
    "- backward pass\n",
    "- 更新模型参数\n",
    "- 每隔一定的iteration输出模型在当前iteration的loss，以及在验证数据集上做模型的评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0, iter: 0, loss: 420.0472106933594\n",
      "epoch: 0, iter: 1000, loss: 110.64915466308594\n",
      "epoch: 0, iter: 2000, loss: 63.69944763183594\n",
      "epoch: 0, iter: 3000, loss: 55.475433349609375\n",
      "epoch: 0, iter: 4000, loss: 45.3078498840332\n",
      "epoch: 0, iter: 5000, loss: 41.215545654296875\n",
      "epoch: 0, iter: 6000, loss: 37.93198776245117\n",
      "epoch: 0, iter: 7000, loss: 35.823787689208984\n",
      "epoch: 0, iter: 8000, loss: 35.2523193359375\n",
      "epoch: 0, iter: 9000, loss: 35.85188674926758\n",
      "epoch: 0, iter: 10000, loss: 35.528446197509766\n",
      "epoch: 0, iter: 11000, loss: 34.1132698059082\n",
      "epoch: 0, iter: 12000, loss: 34.4886474609375\n",
      "epoch: 0, iter: 13000, loss: 33.55486297607422\n",
      "epoch: 0, iter: 14000, loss: 33.06645202636719\n",
      "epoch: 0, iter: 15000, loss: 35.073768615722656\n",
      "epoch: 0, iter: 16000, loss: 33.03211212158203\n",
      "epoch: 0, iter: 17000, loss: 32.87288284301758\n",
      "epoch: 0, iter: 18000, loss: 32.25053024291992\n",
      "epoch: 0, iter: 19000, loss: 33.15672302246094\n",
      "epoch: 0, iter: 20000, loss: 33.965084075927734\n",
      "epoch: 0, iter: 21000, loss: 31.78493881225586\n",
      "epoch: 0, iter: 22000, loss: 31.21977996826172\n",
      "epoch: 0, iter: 23000, loss: 31.222129821777344\n",
      "epoch: 0, iter: 24000, loss: 32.180294036865234\n",
      "epoch: 0, iter: 25000, loss: 32.11918640136719\n",
      "epoch: 0, iter: 26000, loss: 32.410438537597656\n",
      "epoch: 0, iter: 27000, loss: 31.191852569580078\n",
      "epoch: 0, iter: 28000, loss: 31.6796817779541\n",
      "epoch: 0, iter: 29000, loss: 32.07501220703125\n",
      "epoch: 0, iter: 30000, loss: 31.589771270751953\n",
      "epoch: 0, iter: 31000, loss: 30.55971908569336\n",
      "epoch: 0, iter: 32000, loss: 31.50411605834961\n",
      "epoch: 0, iter: 33000, loss: 32.12269592285156\n",
      "epoch: 0, iter: 34000, loss: 31.799057006835938\n",
      "epoch: 0, iter: 35000, loss: 31.488569259643555\n",
      "epoch: 0, iter: 36000, loss: 32.0762939453125\n",
      "epoch: 0, iter: 37000, loss: 31.718448638916016\n",
      "epoch: 0, iter: 38000, loss: 31.836288452148438\n",
      "epoch: 0, iter: 39000, loss: 30.796913146972656\n",
      "epoch: 0, iter: 40000, loss: 31.26019287109375\n",
      "epoch: 0, iter: 41000, loss: 31.18332290649414\n",
      "epoch: 0, iter: 42000, loss: 31.035846710205078\n",
      "epoch: 0, iter: 43000, loss: 31.67388153076172\n",
      "epoch: 0, iter: 44000, loss: 30.95718002319336\n",
      "epoch: 0, iter: 45000, loss: 31.227718353271484\n",
      "epoch: 0, iter: 46000, loss: 30.958019256591797\n",
      "epoch: 0, iter: 47000, loss: 30.802886962890625\n",
      "epoch: 0, iter: 48000, loss: 31.316438674926758\n",
      "epoch: 0, iter: 49000, loss: 31.068218231201172\n",
      "epoch: 0, iter: 50000, loss: 30.90799331665039\n",
      "epoch: 0, iter: 51000, loss: 30.834815979003906\n",
      "epoch: 0, iter: 52000, loss: 30.801921844482422\n",
      "epoch: 0, iter: 53000, loss: 31.455249786376953\n",
      "epoch: 0, iter: 54000, loss: 31.085723876953125\n",
      "epoch: 0, iter: 55000, loss: 31.31935691833496\n",
      "epoch: 0, iter: 56000, loss: 31.21722412109375\n",
      "epoch: 0, iter: 57000, loss: 31.02591323852539\n",
      "epoch: 0, iter: 58000, loss: 30.76601791381836\n",
      "epoch: 0, iter: 59000, loss: 30.669824600219727\n",
      "epoch: 0, iter: 60000, loss: 31.46889305114746\n",
      "epoch: 0, iter: 61000, loss: 30.514265060424805\n",
      "epoch: 0, iter: 62000, loss: 30.749996185302734\n",
      "epoch: 0, iter: 63000, loss: 31.39304542541504\n",
      "epoch: 0, iter: 64000, loss: 31.034055709838867\n",
      "epoch: 0, iter: 65000, loss: 30.75653839111328\n",
      "epoch: 0, iter: 66000, loss: 31.44009780883789\n",
      "epoch: 0, iter: 67000, loss: 30.631933212280273\n",
      "epoch: 0, iter: 68000, loss: 30.852642059326172\n",
      "epoch: 0, iter: 69000, loss: 30.440444946289062\n",
      "epoch: 0, iter: 70000, loss: 30.89693260192871\n",
      "epoch: 0, iter: 71000, loss: 30.34482192993164\n",
      "epoch: 0, iter: 72000, loss: 30.580995559692383\n",
      "epoch: 0, iter: 73000, loss: 30.62125015258789\n",
      "epoch: 0, iter: 74000, loss: 31.25283432006836\n",
      "epoch: 0, iter: 75000, loss: 31.340688705444336\n",
      "epoch: 0, iter: 76000, loss: 31.031070709228516\n",
      "epoch: 0, iter: 77000, loss: 30.721107482910156\n",
      "epoch: 0, iter: 78000, loss: 31.15871238708496\n",
      "epoch: 0, iter: 79000, loss: 30.72861099243164\n",
      "epoch: 0, iter: 80000, loss: 30.686187744140625\n",
      "epoch: 0, iter: 81000, loss: 30.96906280517578\n",
      "epoch: 0, iter: 82000, loss: 30.64755630493164\n",
      "epoch: 0, iter: 83000, loss: 30.86237907409668\n",
      "epoch: 0, iter: 84000, loss: 31.073986053466797\n",
      "epoch: 0, iter: 85000, loss: 30.927490234375\n",
      "epoch: 0, iter: 86000, loss: 30.70050048828125\n",
      "epoch: 0, iter: 87000, loss: 31.14577865600586\n",
      "epoch: 0, iter: 88000, loss: 30.773733139038086\n",
      "epoch: 0, iter: 89000, loss: 30.837783813476562\n",
      "epoch: 0, iter: 90000, loss: 30.594697952270508\n",
      "epoch: 0, iter: 91000, loss: 30.5462646484375\n",
      "epoch: 0, iter: 92000, loss: 31.061866760253906\n",
      "epoch: 0, iter: 93000, loss: 30.809398651123047\n",
      "epoch: 0, iter: 94000, loss: 30.77594757080078\n",
      "epoch: 0, iter: 95000, loss: 30.586809158325195\n",
      "epoch: 0, iter: 96000, loss: 30.77543830871582\n",
      "epoch: 0, iter: 97000, loss: 30.87103271484375\n",
      "epoch: 0, iter: 98000, loss: 30.16326141357422\n",
      "epoch: 0, iter: 99000, loss: 30.258590698242188\n",
      "epoch: 0, iter: 100000, loss: 30.997915267944336\n",
      "epoch: 0, iter: 101000, loss: 30.836084365844727\n",
      "epoch: 0, iter: 102000, loss: 30.87259292602539\n",
      "epoch: 0, iter: 103000, loss: 30.514392852783203\n",
      "epoch: 0, iter: 104000, loss: 30.22088050842285\n",
      "epoch: 0, iter: 105000, loss: 30.869644165039062\n",
      "epoch: 0, iter: 106000, loss: 30.51873016357422\n",
      "epoch: 0, iter: 107000, loss: 30.863828659057617\n",
      "epoch: 0, iter: 108000, loss: 30.66390609741211\n",
      "epoch: 0, iter: 109000, loss: 30.905582427978516\n",
      "epoch: 0, iter: 110000, loss: 31.3685302734375\n",
      "epoch: 0, iter: 111000, loss: 30.87140655517578\n",
      "epoch: 0, iter: 112000, loss: 30.823673248291016\n",
      "epoch: 0, iter: 113000, loss: 31.092639923095703\n",
      "epoch: 0, iter: 114000, loss: 30.361209869384766\n",
      "epoch: 0, iter: 115000, loss: 30.724641799926758\n",
      "epoch: 0, iter: 116000, loss: 30.91895866394043\n",
      "epoch: 0, iter: 117000, loss: 30.906103134155273\n",
      "epoch: 0, iter: 118000, loss: 30.354206085205078\n",
      "epoch: 0, iter: 119000, loss: 30.33993911743164\n",
      "epoch: 1, iter: 0, loss: 30.718202590942383\n",
      "epoch: 1, iter: 1000, loss: 30.54977035522461\n",
      "epoch: 1, iter: 2000, loss: 30.735172271728516\n",
      "epoch: 1, iter: 3000, loss: 30.505659103393555\n",
      "epoch: 1, iter: 4000, loss: 30.68982696533203\n",
      "epoch: 1, iter: 5000, loss: 30.236034393310547\n",
      "epoch: 1, iter: 6000, loss: 30.450191497802734\n",
      "epoch: 1, iter: 7000, loss: 30.46800994873047\n",
      "epoch: 1, iter: 8000, loss: 30.587671279907227\n",
      "epoch: 1, iter: 9000, loss: 30.89776039123535\n",
      "epoch: 1, iter: 10000, loss: 30.453964233398438\n",
      "epoch: 1, iter: 11000, loss: 30.340309143066406\n",
      "epoch: 1, iter: 12000, loss: 30.579856872558594\n",
      "epoch: 1, iter: 13000, loss: 30.941007614135742\n",
      "epoch: 1, iter: 14000, loss: 30.401355743408203\n",
      "epoch: 1, iter: 15000, loss: 30.470251083374023\n",
      "epoch: 1, iter: 16000, loss: 30.4851131439209\n",
      "epoch: 1, iter: 17000, loss: 31.073944091796875\n",
      "epoch: 1, iter: 18000, loss: 30.842205047607422\n",
      "epoch: 1, iter: 19000, loss: 30.65728759765625\n",
      "epoch: 1, iter: 20000, loss: 30.513702392578125\n",
      "epoch: 1, iter: 21000, loss: 30.401121139526367\n",
      "epoch: 1, iter: 22000, loss: 30.139076232910156\n",
      "epoch: 1, iter: 23000, loss: 30.591323852539062\n",
      "epoch: 1, iter: 24000, loss: 30.598430633544922\n",
      "epoch: 1, iter: 25000, loss: 30.648696899414062\n",
      "epoch: 1, iter: 26000, loss: 31.000125885009766\n",
      "epoch: 1, iter: 27000, loss: 30.409883499145508\n",
      "epoch: 1, iter: 28000, loss: 30.217910766601562\n",
      "epoch: 1, iter: 29000, loss: 30.617969512939453\n",
      "epoch: 1, iter: 30000, loss: 30.025922775268555\n",
      "epoch: 1, iter: 31000, loss: 30.6910457611084\n",
      "epoch: 1, iter: 32000, loss: 30.752849578857422\n",
      "epoch: 1, iter: 33000, loss: 30.63628578186035\n",
      "epoch: 1, iter: 34000, loss: 30.728408813476562\n",
      "epoch: 1, iter: 35000, loss: 30.473583221435547\n",
      "epoch: 1, iter: 36000, loss: 30.26988410949707\n",
      "epoch: 1, iter: 37000, loss: 30.499309539794922\n",
      "epoch: 1, iter: 38000, loss: 30.24459457397461\n",
      "epoch: 1, iter: 39000, loss: 30.781661987304688\n",
      "epoch: 1, iter: 40000, loss: 30.572044372558594\n",
      "epoch: 1, iter: 41000, loss: 30.40477752685547\n",
      "epoch: 1, iter: 42000, loss: 29.991519927978516\n",
      "epoch: 1, iter: 43000, loss: 30.25002098083496\n",
      "epoch: 1, iter: 44000, loss: 30.90960693359375\n",
      "epoch: 1, iter: 45000, loss: 30.67990493774414\n",
      "epoch: 1, iter: 46000, loss: 30.910268783569336\n",
      "epoch: 1, iter: 47000, loss: 30.526987075805664\n",
      "epoch: 1, iter: 48000, loss: 29.802139282226562\n",
      "epoch: 1, iter: 49000, loss: 30.346729278564453\n",
      "epoch: 1, iter: 50000, loss: 30.558292388916016\n",
      "epoch: 1, iter: 51000, loss: 30.506786346435547\n",
      "epoch: 1, iter: 52000, loss: 30.17622947692871\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1, iter: 53000, loss: 30.422870635986328\n",
      "epoch: 1, iter: 54000, loss: 30.511688232421875\n",
      "epoch: 1, iter: 55000, loss: 30.220746994018555\n",
      "epoch: 1, iter: 56000, loss: 30.517898559570312\n",
      "epoch: 1, iter: 57000, loss: 30.228534698486328\n",
      "epoch: 1, iter: 58000, loss: 29.6726016998291\n",
      "epoch: 1, iter: 59000, loss: 30.818809509277344\n",
      "epoch: 1, iter: 60000, loss: 30.15318489074707\n",
      "epoch: 1, iter: 61000, loss: 30.516082763671875\n",
      "epoch: 1, iter: 62000, loss: 30.354167938232422\n",
      "epoch: 1, iter: 63000, loss: 30.65067481994629\n",
      "epoch: 1, iter: 64000, loss: 30.16307830810547\n",
      "epoch: 1, iter: 65000, loss: 30.29519271850586\n",
      "epoch: 1, iter: 66000, loss: 30.450511932373047\n",
      "epoch: 1, iter: 67000, loss: 30.45915985107422\n",
      "epoch: 1, iter: 68000, loss: 30.23889923095703\n",
      "epoch: 1, iter: 69000, loss: 30.60623550415039\n",
      "epoch: 1, iter: 70000, loss: 30.522491455078125\n",
      "epoch: 1, iter: 71000, loss: 30.621078491210938\n",
      "epoch: 1, iter: 72000, loss: 30.627422332763672\n",
      "epoch: 1, iter: 73000, loss: 30.85080337524414\n",
      "epoch: 1, iter: 74000, loss: 30.2876033782959\n",
      "epoch: 1, iter: 75000, loss: 30.20944595336914\n",
      "epoch: 1, iter: 76000, loss: 30.489473342895508\n",
      "epoch: 1, iter: 77000, loss: 30.41691017150879\n",
      "epoch: 1, iter: 78000, loss: 30.45703887939453\n",
      "epoch: 1, iter: 79000, loss: 30.2554874420166\n",
      "epoch: 1, iter: 80000, loss: 30.326425552368164\n",
      "epoch: 1, iter: 81000, loss: 30.35062599182129\n",
      "epoch: 1, iter: 82000, loss: 31.12557029724121\n",
      "epoch: 1, iter: 83000, loss: 30.172191619873047\n",
      "epoch: 1, iter: 84000, loss: 30.336368560791016\n",
      "epoch: 1, iter: 85000, loss: 30.32604217529297\n",
      "epoch: 1, iter: 86000, loss: 30.477481842041016\n",
      "epoch: 1, iter: 87000, loss: 30.725988388061523\n",
      "epoch: 1, iter: 88000, loss: 30.35128402709961\n",
      "epoch: 1, iter: 89000, loss: 31.01091766357422\n",
      "epoch: 1, iter: 90000, loss: 30.491222381591797\n",
      "epoch: 1, iter: 91000, loss: 30.21773910522461\n",
      "epoch: 1, iter: 92000, loss: 29.86251449584961\n",
      "epoch: 1, iter: 93000, loss: 29.85382652282715\n",
      "epoch: 1, iter: 94000, loss: 30.321523666381836\n",
      "epoch: 1, iter: 95000, loss: 29.814773559570312\n",
      "epoch: 1, iter: 96000, loss: 30.574665069580078\n",
      "epoch: 1, iter: 97000, loss: 30.14328956604004\n",
      "epoch: 1, iter: 98000, loss: 29.90842056274414\n",
      "epoch: 1, iter: 99000, loss: 30.273555755615234\n",
      "epoch: 1, iter: 100000, loss: 29.937362670898438\n",
      "epoch: 1, iter: 101000, loss: 30.375999450683594\n",
      "epoch: 1, iter: 102000, loss: 30.64742660522461\n",
      "epoch: 1, iter: 103000, loss: 30.526161193847656\n",
      "epoch: 1, iter: 104000, loss: 30.345212936401367\n",
      "epoch: 1, iter: 105000, loss: 30.510639190673828\n",
      "epoch: 1, iter: 106000, loss: 30.42151641845703\n",
      "epoch: 1, iter: 107000, loss: 30.565765380859375\n",
      "epoch: 1, iter: 108000, loss: 29.951072692871094\n",
      "epoch: 1, iter: 109000, loss: 30.22141456604004\n",
      "epoch: 1, iter: 110000, loss: 30.723838806152344\n",
      "epoch: 1, iter: 111000, loss: 30.395931243896484\n",
      "epoch: 1, iter: 112000, loss: 30.433988571166992\n",
      "epoch: 1, iter: 113000, loss: 30.041824340820312\n",
      "epoch: 1, iter: 114000, loss: 30.053401947021484\n",
      "epoch: 1, iter: 115000, loss: 30.419116973876953\n",
      "epoch: 1, iter: 116000, loss: 30.211109161376953\n",
      "epoch: 1, iter: 117000, loss: 30.66413116455078\n",
      "epoch: 1, iter: 118000, loss: 29.81429672241211\n",
      "epoch: 1, iter: 119000, loss: 30.27712059020996\n"
     ]
    }
   ],
   "source": [
    "\n",
    "for e in range(EPOCHS):\n",
    "    for i, (input_labels, pos_labels, neg_labels) in enumerate(dataloader):\n",
    "        # TODO\n",
    "        input_labels = input_labels.long()\n",
    "        pos_labels = pos_labels.long()\n",
    "        neg_labels = neg_labels.long()\n",
    "\n",
    "        input_labels = input_labels.to(device)\n",
    "        pos_labels = pos_labels.to(device)\n",
    "        neg_labels = neg_labels.to(device)\n",
    "            \n",
    "        optimizer.zero_grad()\n",
    "        loss = model(input_labels, pos_labels, neg_labels).mean()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        if i % 1000 == 0:\n",
    "            print(\"epoch: {}, iter: {}, loss: {}\".format(e, i, loss.item()))\n",
    "            \n",
    "        \n",
    "#         if i % 2000 == 0:\n",
    "#             embedding_weights = model.input_embeddings()\n",
    "#             sim_simlex = evaluate(\"simlex-999.txt\", embedding_weights)\n",
    "#             sim_men = evaluate(\"men.txt\", embedding_weights)\n",
    "#             sim_353 = evaluate(\"wordsim353.csv\", embedding_weights)\n",
    "#             with open(LOG_FILE, \"a\") as fout:\n",
    "#                 print(\"epoch: {}, iteration: {}, simlex-999: {}, men: {}, sim353: {}, nearest to monster: {}\\n\".format(\n",
    "#                     e, i, sim_simlex, sim_men, sim_353, find_nearest(\"monster\")))\n",
    "#                 fout.write(\"epoch: {}, iteration: {}, simlex-999: {}, men: {}, sim353: {}, nearest to monster: {}\\n\".format(\n",
    "#                     e, i, sim_simlex, sim_men, sim_353, find_nearest(\"monster\")))\n",
    "                \n",
    "    embedding_weights = model.input_embeddings()\n",
    "    np.save(\"embedding-{}\".format(EMBEDDING_SIZE), embedding_weights)\n",
    "    torch.save(model.state_dict(), \"embedding-{}.th\".format(EMBEDDING_SIZE))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 未完待续\n",
    " 需要明白的点：\n",
    "    采样\n",
    "    输入输出的embedding是分开的？\n",
    "    后面的验证是什么鬼？\n",
    "    如何评价？\n",
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
